Application and network abuse detection with adaptive mitigation utilizing multi-modal intelligence data

ABSTRACT

In an embodiment, a computer-implemented method detects a network or application abuse to a service provider environment. In the method, data is collected describing incoming requests from plurality of different external source addresses to the service provider environment. The collected data is used to compare the incoming requests against a heuristic. When the incoming requests are determined to match the heuristic, the requests, having the plurality of different external source addresses, are from a common abuse entity. Finally, the collected data is evaluated to determine that the common abuse entity is a potential network abuser of the service provider environment.

BACKGROUND

1. Field

This field is generally related to network security.

2. Related Art

A communication network may, for example, allow data to be transferredbetween two geographically remote locations. Networks are used, forexample, to provide applications, such as web and other Internet-basedapplications, to users. Typically, these applications operate byreceiving a request, such as an Hypertext Transfer Protocol (HTTP)request, and, based on the request, supplying a response. The requestand response may be formatted in accordance with a known applicationprogram interface (application). The requests are generally transmittedvia a public or private network, such as the Internet or an internalnetwork, to the service provider. The service provider has its ownenvironment that services the request. The environment may include aplurality of different devices that coordinate with each other toprovide the service. The devices may coordinate over a private networkbelonging to the service provider. Or, the devices may operate in acloud or a public network.

Not all application and network requests are legitimate. Often times,these requests are meant to abuse the network or the application. Abusecan come in several forms. For example, some abuse mechanisms try tooverwhelm a service so that it cannot service legitimate requests. Theseare referred to as denial of service requests, whether at the network orapplication layer. One common mechanism of abuse is referred to asapplication abuse. An example of this is an a malicious entityfraudulently creating accounts on a service provider platform and thentransport unwanted requests across the service provider environment.

Another type of denial of service abuse is a Transport Control Protocol(TCP) SYN flood abuse. Normally when a client attempts to start a TCPconnection to a server, the client requests a connection by sending aSYN (synchronize) message to the server, the server acknowledges thisrequest by sending SYN-ACK back to the client, and the client respondswith an ACK. A SYN flood abuse works by not responding to the serverwith the expected ACK code, failing to finish the transaction. Enough ofthese unfinished transactions can overwhelm a server, rendering itunable to respond to additional requests.

Other abuses may not be trying to bring down a service, but may insteadbe making requests for other improper purposes. In these abuses, anautomated system may be making application requests that, for example,set up fake user accounts and try to entice a user to devolveconfidential information, such as her password, credit card information,or Social Security number, or run other scams. These abuses aresometimes referred to as application or application abuse. Often times,these abuse vectors can be concealed inside of an encrypted transportmethod, such as SSL (Secure Sockets Layer) or IPSec (Internet ProtocolSecurity).

Hardware appliances are available that try to control these type ofnetwork and application abuses. Some of these appliances may, forexample, operate by maintaining a database of fingerprints of knownthreats. A database of known threats may be generated by human analystsand include fingerprints identifying different potential threats. As theappliance manufacturer becomes aware of new threats, it may send updatesto the database. Using the database, the appliance scans for potentialthreats.

In addition to scanning against fingerprints, some appliances may checkthe rate of requests from particular source addresses. For example, anappliance may recognize that requests from a source address increasedramatically or exceed a threshold to detect potential abuses.

While these appliances have advantages, they suffer at least threeprimary drawbacks. First, they may be impossible to deploy in particulararchitectures, such as some cloud applications hosted by third parties.Second, they tend to operate in their own silos, often consisting onlyof their customer network and application transaction data to updatethreat databases. Operating in their own silos, these appliances may noteffectively adapt and react to new threats. Third, they tend to bepurpose-built for only a narrow class of abuse.

Limited in these respects, some malicious entities can spread theirrequests out from a variety of different source addresses and circumventthese security measures. New systems and methods are needed to betterprotect against these abuses.

BRIEF SUMMARY

In an embodiment, a computer-implemented method detects a network abuseto a service provider environment. In the method, data is collecteddescribing incoming requests from a plurality of different externalsource addresses to the service provider environment. The collected datais used to compare the incoming requests against a heuristic. When theincoming requests are determined to match the heuristic, the requests,having the plurality of different external source addresses, are from acommon source. Finally, the collected data is evaluated to determinethat the common source is a potential network abuser of the serviceprovider environment.

System and computer program product embodiments are also disclosed.

Further embodiments, features, and advantages of the invention, as wellas the structure and operation of the various embodiments, are describedin detail below with reference to accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated herein and form partof the specification, illustrate the present disclosure and, togetherwith the description, further serve to explain the principles of thedisclosure and to enable a person skilled in the relevant art to makeand use the disclosure.

FIG. 1A is a diagram illustrating a system for abuse detection andmitigation, according to an embodiment.

FIGS. 1B-C illustrate additional configurations and features.

FIG. 2 is a diagram illustrating components of the system in FIG. 1 thatcollect state from a service provider network for threat detection,according to an embodiment.

FIG. 3 is a diagram illustrating components of the system in FIG. 1

FIG. 4 is a flowchart illustrating a method for abuse detection,according to an embodiment.

The drawing in which an element first appears is typically indicated bythe leftmost digit or digits in the corresponding reference number. Inthe drawings, like reference numbers may indicate identical orfunctionally similar elements.

DETAILED DESCRIPTION

As mentioned above, operating in their own silos, existing networkappliances may not effectively adapt and react to new denial of networkand application abuse threats. To deal with this, embodiments use datafrom the service provider environments in concert with external threatdata and/or historical data to identify malicious activity, even whenusing multiple vectors of abuse. More specifically, embodiments collectdata on requests to the service provider's environment. They compare therequests against a set of heuristics to determine whether the differentrequests, being transmitted from or received from different entities,may, in fact be from a common abuse entity. The data is also evaluatedagainst multiple data heuristics to determine whether the source(s) maybe attempting or actively abusing the service. If application or networkabuse is determined to be in progress, an operator may be alerted orsteps to mitigate the abuse may be provided to the administrator orsystem for manual or automatic mitigation.

FIG. 1 is a diagram illustrating a system 100 for abuse detection andmitigation, according to an embodiment. System 100 includes one or morenetwork connected entities 102, such as the Internet, and a serviceprovider environment 108. System 100 also includes an attack mitigationdevice 106, a threat detection device 120 and an external threat dataprovider(s) 130. Each of these components is described below, and inmore detail with respect to FIGS. 2 and 3.

Network connected entities 102 includes a plurality of abuse resources104. Abuse resources 104 may be a number of different devices withdifferent identities. For example, abuse resources 104 may beaddressable on network connected entities 102 by differing InternetProtocol (IP) addresses or other resource identifiers, such as HTTPUser-Agents, DNS data, IP routing information, reputation data, etc.

Abuse resources 104 may be computers of or controlled by a maliciousperson, such as a malicious entity. For example, they may be computingdevices that the abuse resource owns, or at least partially controls,for the purpose of enacting harm upon the service provider environmentor users thereof. The malicious entity can highjack devices 104 to takepart in an abuse by installing a virus or malware. For example, in theSYN abuse described above, the malicious entity can engage a number ofdifferent devices 104 to initiated uncompleted TCP sessions by infectingthe devices with malware. Or, the malicious entity can engage devices104 to take part in the abuse using their own call-response protocol.For example, the malicious entity can engage devices 104 to take part inthe abuse by sending messages with a fraudulent return address,prompting the devices to reply to the fraudulent return address, whichcan overwhelm it.

In system 100, abuse resources 104 may be engaged in an abuse of serviceprovider network 108. As discussed above, the abuse may be a denial ofservice attack targeted at making the service provider unable to servicelegitimate requests or may be an API abuse that leverages the serviceprovider's APIs for an illegitimate purpose.

Service provider environment 108 may include servers designed to serviceapplication requests either on the local service provider environment ormultiple service providers associated with the main service providerenvironment. These service providers may be upstream Internet Providers,Cloud Providers or other services and resources the service providerenvironment is a customer thereof. Service provider environment 108 maybe the environment of a customer of the threat detection device 120. Theservers may be reachable from network connected entities 102 usingparticular destination addresses, such as source or destination IPaddresses or hostnames. The IP addresses may be registered on a nameservice, such as a Domain Name System (DNS) service. Using the DNSservice, computing devices may look up the appropriate IP addressesbased on a hostname and vice versa.

The servers in service provider environment 108 may be configured toservice application requests formatted in particular manner. Thefunction of the application requests can vary widely depending on theservice provided. In an example where the service is a social mediaservice, the application requests may create new user accounts, requestdata, post data to the user's account, or post data to other user'saccounts, enabling various users on the social media service tocommunicate.

According to an embodiment, threat detection device 120 detects on theservice provider environment 108 abuses using service provider networkdata 110 collected from service provider environment 108 and using athread feed 132 from external threat data provider 130.

Threat detection device 120 includes a source recognition module 122 anda mitigation module 124. Source recognition module 122 determines that acommon abuse entity controls abuse resources 104. Abuse resources 104may try to conceal that they are from a common abuse entity by, forexample, using different source addresses, such as different source IPaddresses. As discussed in greater detail below, to determine whether acommon abuse entity controls the abuse resources 104, source recognitionmodule may evaluate customer or service provider environment data 110against a heuristic. Threat detection device 120 may further evaluatecustomer or service provider environment data 110 to determine that thecommon abuse entity is engaged in an abuse

If threat detection device 120 determines that the common abuse entitycontrolling abuse resources 104 is engaged in a potential abuse,mitigation module 124 determines what, if anything, should be tomitigate it. Mitigation module 124 may specify any mitigation actions onmitigation instructions 142 and send mitigation instructions 142 toattack mitigation device 106.

Attack mitigation device 106 may, for example, be a commonly availableor specialized firewall, router, switch, load balancer, distributeddenial of service (DDOS) mitigation appliance or other devices tomitigate the abuse. When attack mitigation device 106 receivesmitigation instructions 142, it takes action to mitigate the abuse. Thismay mean blocking certain traffic, such as traffic having certain sourceIP addresses, or marking certain user accounts as suspect due toanomalous application behavior or threat indicators.

In addition to mitigating attacks, attack mitigation device 106 maycollect data about incoming requests, like customer environment data 110from service provider network 108. The collected data may be sent tothreat detection device 120, which uses the data to detect potentialnetwork abuses.

In this way, embodiments detect and mitigate network abuses usingservice provider environment data. Additional configurations andfeatures are illustrated in FIGS. 1B-C. And other aspects are describedin greater detail below with respect to FIGS. 2 and 3.

FIG. 1B shows a system 150 illustrating an embodiment where threatdetection device 120 uses data on decrypted requests to detect an abuseand attack mitigation device 108 can block still-encrypted requests.

In system 150, service provider environment 108 includes a decryptionserver 162 and an API server 164. API server 164 services applicationrequests. Application requests are often encrypted, for example, using aSecure Sockets Layer (SSL) technology. Decryption server 162 may be anSSL server that decrypts the incoming requests before they reach APIserver 164. This relieves API server 164 from conducting decryptiontasks, which can be computationally expensive.

Because attack mitigation device 106 is upstream from decryption server162, when network connected entities 102 send an encrypted request 154,encrypted request 154 may reach attack mitigation device 106 beforedecryption server 162. Attack mitigation device 106 may evaluate therequest packet's headers (at, for example, network layers two or three)to determine whether to block or otherwise mitigate the incoming packet.But, because request 154's payload (at, for example, the applicationlayer) is encrypted, attack mitigation device 106 may be unable toeffectively evaluate its payload. Absent mitigation instructionsidentifying the encrypted request 154 as suspect, attack mitigationdevice 106 may forward the encrypted request 154 onto decryption server162.

Decryption server 162 decrypts encrypted request 154 and forwards ontoAPI server 164 as decrypted request 156. API server 164 may collect dataon decrypted request 156, including its payload, and send the data ontothreat detection device 120 as customer environment data 110. Thedecrypted payload data sent to threat detection device 120 may includeinformation about the application-level API call that would be difficultto collect were the request encrypted. Using this data, threat detectiondevice 120 can make more informed determinations about a potentialthreat than could be accomplished with a single appliance only lookingat encrypted data.

When threat detection device 120 detects a potential threat, it can sendmitigation instructions 142 that inform attack mitigation device 106 toblock or otherwise mitigate incoming packets having particular headercharacteristics, such as network layers two or three headercharacteristics. For example, mitigation instructions 142 may identifyparticular source IP addresses of suspect packets and instruct attackmitigation device 106 to block those packets. In this way, using threatdetection device 120, attack mitigation device 106 can block a potentialabuse that may be otherwise undetectable from its position on thenetwork upstream from decryption server 162.

FIG. 1C shows a system 180 illustrating an embodiment where threatdetection device 120 uses knowledge of network topology to select one ofseveral attack mitigation devices to mitigate an attack.

System 180 includes two service provider environments—108A and 108B—andtwo attack mitigation devices-106A and 106B. As mentioned above, eachservice provider environment 108A-B can include its own network, andthat network can forward data onto other connected devices and networks.Network connected entities 102 connects to downstream attack mitigationdevice 106A, which connects to downstream service provider environment108A, which connects to downstream attack mitigation device 106B, whichconnects to downstream service provider environment 108B.

According to an embodiment, threat detection device 120 is aware of thistopology and selects where to send mitigation instructions (to attackmitigation device 106A or 106B) based on its knowledge of the topology.Threat detection device 120 may determine the targets of the abuse andmay send mitigation instructions to a mitigation device upstream fromthe targets. In one embodiment, threat detection device 120 may select amitigation device as close to the targets as possible while still beingupstream from the targets. For example, if a threat is only attackingservice provider environment 108B, threat detection device 120 maymitigate the attack by sending mitigation instructions 142B to attackmitigation device 106B. Alternatively, if a threat is attacking bothservice provider environment 108A and service provider environment 108B,threat detection device 120 may mitigate the attack by sendingmitigation instructions 142A to attack mitigation device 106A. In thisway, threat detection device 120 can efficiently balance mitigationresponsibility across a number of devices in a network.

While only two devices and environments are shown in the illustrativeexample in system 180, a skilled artisan would recognize other devicesand environments may be used in a similar manner.

FIG. 2 is a diagram illustrating a system 200 that collects data from aservice provider network for threat detection. Like system 100 in FIG.1, system 200 includes service provider network 108 and threat detectiondevice 120.

Threat detection device 120 includes a collection module 210 thatcollects data from service provider network 108. It collects differenttypes of data from different devices. The data may reflect incomingrequests to service provider network 108. The requests may have data atdifferent network levels in the Open Systems Interconnection (OSI)network hierarchy. The requests, for example, may be application-levelrequests or network-level requests. And, to observe the differentlevels, collection module 210 may need to access different devices onservice provider network 108.

Service provider network 108 has network devices 202, API servers 206,and name servers 204. Network devices 202 may, for example, be switchingdevices that route a packet from one port of the device to another basedon the packet's destination address. The switching devices may operateat layer 2 and the packet may be addressed using a Media Access Control(MAC) address. One widespread example of a layer 2 networking protocolis Ethernet. The switching devices may also operate at a layer 3. Thesedevices, commonly called routers, may route data based on an IP address.The data collected from network devices 202 may, for example, be netflowdata or packet capture (PCAP) data.

Netflow data, as the term is used herein, is not limited to data from aparticular brand or type of router. The netflow data may include arecord for each data flow. Each data flow may be one or more packets intime proximity with one another having a common protocol, and commonabuse entity identified via Internet Protocol (IP) addresses andTransport Control Protocol (TCP) or User Datagram Protocol (UDP) ports.When a certain amount of time passes after receipt of a packet havingthese characteristics, the network device determines that the flow hasended, and if the network device receives any additional packets withthese characteristics, the network device regards the packets asbelonging to a new data flow and represents them with a new netflow datarecord. Each netflow record may include the data flow's (1) source anddestination IP address, (2) source port and destination UDP or TCP port,(3) type of protocol, (4) start and end times, and (5) size (e.g.,number of bytes). In this way, netflow data summarizes certaincharacteristics of a data flow.

Unlike this summary netflow information, packet inspection or packetcapture (PCAP) data can capture an entire packet, and/or create a recordof the details of an application or data flow. This may be useful forinspecting the body and payload of a packet and its contents. Networkdevices 202 may have operating system interfaces that enable thisfeature. Collecting all packets in this method may be too costly onnetwork and computing resources thus, collection module 210 may samplethis data, perhaps only capturing the first packet, or first severalpackets, in each data flow.

While collection module 210 collects protocol information from thevarious network layers from network devices 202, collection module 210collects data from the application layer from API servers 206. Theapplication layer data may include data in the application requests. Inthe social media service example above, if the application requests areto create new user accounts, the data collected may include the desireduser name, password, user-agent, timestamp and source IP address. If theapplication request is to post new data to the user's account or toanother user's account, collection module 210 may collect the datasought to be posted. Further, in the embodiments, data can be obtainedfrom application and IT infrastructure monitoring solutions includinglogging of various types, syslog, application and programming codemanagement and performance monitoring systems instrumentation andoutput, and others which could singularly or collectively, provideintelligence data to identify abuse in a service providerenvironment(s).

Finally, name server 204 may be a Domain Name System server. Consistentwith functionality of other external name servers, name server 204 maymap hostnames to particular IP addresses and vice versa. In a specificexample, name server 204 may map hostnames within the service providerserver's domain to specific IP addresses. If a service provider hasregistered the domain example.com, name server 204 may map hostnameswithin that domain (e.g., matching *.example.com) to particular IPaddresses. From name server 204, collection module 210 may collect datadescribing hostname lookups within the service provider's domain.

To collect data from network devices 202, API servers 206 and nameservers 204, collection module 210 may periodically send requests tothose respective devices. Alternatively, those devices may push datadirectly to collection module 210. The devices may have applicationsalready installed that allow collection module 210 to collect the data.Alternatively an additional module may have to be installed on thosedevices to observe their behavior and send data back to collectionmodule 210.

In this way, embodiments collect data describing incoming requests intothe service provider's environment. Once the data is collected, it isevaluated to detect potential abuses as illustrated in FIG. 3.

FIG. 3 is a diagram illustrating a threat detection system 300. In anexample, the functionality of system 300 in FIG. 3 may be included inthreat detection device 120 in FIG. 1. Like threat detection device 120,system 300 includes source recognition module 122, which is included ina machine learning module 330, and mitigation module 124, and receivesservice provider environment data 110 and threat feed 132. In addition,system 300 includes an aggregation module 310 and a threat recognitionmodule 316. And, system 300 includes network data 302, heuristics 304,threat rules 306, and mitigation rules 308, which are all accessibleusing an administrative interface module 318. Each of these componentsis addressed in turn.

The amount of data collected in the manner described for FIG. 2 can getlarge quickly. For this reason, aggregation module 310 encodes customeror service provider environment data 110 so that it requires less spacebefore storing it into network data 302. In one embodiment, aggregationmodule 310 may aggregate the collected data into counts of requestshaving common characteristics. For example, if five requests were sentfrom a particular source address in a day and each request was in adifferent data flow, the netflow data may have five records: one foreach data flow. As described above, each record may include a start andend times (or a duration) of the data flow. Aggregation module 310 mayaggregate the five records into one stating that on that day a total offive requests were received from that source address. For applicationdata, aggregation module 310 can aggregate in a similar manner. Forexample, if a particular source address makes five application calls tocreate new user accounts, aggregation module 310 may aggregate the fiverecords into one stating that on that day a total of five new userrequests were received from that source address.

In addition, data from the different sources or destinations on thecustomer network may be correlated with one another to reduce oreliminate repetitive data. For example, an application request mayappear both as netflow data and as an application request. These recordsmay be combined so that the source address and time of receipt onlyneeds to be stored once. Outside of these specific examples, aggregationmodule 310 may use other de-duplication and encoding techniques generatede-duplicated data 312 and store it in network data 302.

Machine learning module 330 uses network data 302 to detect new abusemechanisms. A combination of IP source and destination, applicationtransactional, DNS transactions, network communication patterns, inconcert with external data sources will provide the machine learningmodule 330 with the ability to more accurately detect threats. Inparticular, machine learning module 330 uses the following data togenerate a decision: (i) Service provider environment data; includingnetwork, DNS and application transaction requests; (ii) Externalreputation, and threat, and vulnerability data and historical DNStransactions from providers; and (iii) Data from network and systemssecurity appliances or software within the service provider environment.

Machine learning module 330 uses machine learning techniques to detectand respond to these malicious requests. Example machine learningtechniques include decision tree learning, association rule learning,artificial neural networks, inductive logic programming, support vectormachines, clustering, Bayesian networks, reinforcement learning,representation learning, similarity and metric learning, and sparsedictionary learning.

Machine learning techniques generally are trained on a set of learnedservice provider environment data, external data sources, and, aftertraining, reacts to new data based in accordance. For example, machinelearning module 330 may be trained with a set of known circumstancesthat are recognized as being a network abuse and may know how to respondin the event those known circumstances occur. In addition, from thosecircumstances, machine learning module 330 may generate heuristics andrules that can be used to react to other circumstances.

To detect these types abuses, machine learning module 330 includessource recognition module 122 and threat recognition module 316. Sourcerecognition module 122 analyzes the collected data in network data 302to compare the incoming requests against heuristics 304. When aplurality of the incoming requests is determined to match the heuristic,source recognition module determines that the requests, having theplurality of different sources, are from a common abuse entity 314. Inthe example where the respective incoming requests are applicationcalls, the heuristic may be a regular expression rule, or other patternmatching rule, against a field of the application call. In the examplewhere the application call is to create a new user account, the patternmatching rule may be against a requested username. Source recognitionmodule 122 determines whether the application calls match a regularexpression or satisfy the pattern matching rule. When the applicationcalls are determined to match a regular expression or satisfy thepattern matching rule, source recognition module 122 determines that therequests, having a plurality of different external source addresses, arefrom the common abuse entity.

In the example where the data may be netflow data, source recognitionmodule 122 may use other heuristics. For example, the heuristics may mapgroups of IP addresses to certain known actors. These groups of IPaddresses/hostnames may be IP addresses that a service provider assignedto a single person. Or, they may be IP addresses that are assigned inthe DNS system to a single domain. When several requests are receivedfrom source IP addresses in this group, source recognition module 122identifies them as being from a common abuse entity. The groups may alsobe categorized by subnet. In that embodiment, a heuristic may identify aparticular subnet as belonging to a common abuse entity. For example,when several different IP addresses have their first 32 bits in common,source recognition module 122 may identify them has coming from a commonabuse entity.

Once identified, source recognition module 122 sends common abuseentities 314 to threat recognition module 316. Common sources 314 mayidentify a group of IP addresses or application requests belonging toeach common request.

Threat recognition module 316 evaluates network data 302 to determinewhether the common abuse entity is a potential malicious entity of theservice provider environment. To determine whether the common abuseentity is a potential malicious entity, threat recognition module 316can use threat rules 306. Threat rules 306 may specify conditions wherethe threat recognition module 316 identifies a source as a maliciousentity. The conditions may be based on a variety of inputs, including arate, an external threat feed, and others.

In the rate-based approach, threat recognition module 316 evaluates thecollected data to determine a rate of incoming network and applicationrequests from the common abuse entity. Threat recognition module 316 maydetermine whether the rate of incoming requests, having a particulartype (e.g., network or application layer requests and if application,whether it is HTTP or some other protocol) matches a heuristic. The ratemay match a heuristic when it exceeds a threshold specified by theheuristic. The threshold may be a fixed value in threat rules 306 or maybe based on prior traffic, such as prior traffic from the source. Forexample threshold may be a certain number of standard deviations awayfrom rates that were previously measured. And, when the rate of incomingrequests is determined to exceed the threshold, threat recognitionmodule 316 determines that the common abuse entity is a potentialnetwork malicious entity of the service provider environment.

Threat rules 306 may also be based on external threat feed 132. Threatrecognition module 316 receives, from external threat feed 132,fingerprint data identifying a suspect source address and determinesthat the common abuse entity is the potential network malicious entitybased on whether the suspect source address from the external data feedbelongs to the common abuse entity. Fingerprint data may be stored withthreat rules 306 for future use. External threat feed may also includereputation data surrounding different source addresses. The poorreputation data may indicate that others have reported bad conduct ofthe IP address or other network or resource identifier. The externalthreat feed and historical DNS heuristics may also be used as a feedbackmechanism to train new threat rules 306 in machine learning module 330.

The external threat feed may also include data from news sources (suchas a Google News source available from Google Inc. of Mountain ViewCalif.) and social media sources (such as a Facebook source availablefrom Facebook, Inc. of Menlo Park, Calif.). For example, an uprising inthe Middle East may appear as a spike in traffic from a particulargeographic area, which the threat rules would otherwise register as anabuse. But, data from these news or social media sources may indicatethat it is not an abuse but a wave of legitimate traffic caused by areal-world event.

Finally, the external threat feed may include real-time DNS transactiondata. For example, sources that are requesting similar application ornetwork request transactions may be determined as from a common abuseentity. To determine whether sources are requesting similar applicationor network transactions, the DNS transaction data may be used. In theabove examples, in addition DNS heuristics can include the amount oftimes a domain has had key DNS resource records changed (e.g., frequencyof recent changes), their geographic location and any threat data theabove heuristics.

In addition to evaluating a rate of the requests, threat rules 306 mayalso look at other past conduct of the source. For example, in the caseof application abuse, threat rules 306 may indicate a potential threatwhen no prior requests are received from the source and now they arecalling applications in a regular pattern.

Finally, threat recognition module 316 may look to the number of IPaddresses mapped to a particular domain in the Domain Name System, thegeographic origination of source IP addresses, or whether any of theincoming requests has used a fraudulent credit card or having beenassociated with other type of malicious behavior.

Threat recognition module 316 compares this information or anycombination thereof with thresholds and conditions defined in threatrules 306 to determine whether the common abuse entity is a potentialmalicious entity. Threat recognition module 316 then sends itsdetermination to mitigation module 124.

To account these different factors—e.g., external threat data, ratechanges, geographic originating, prior malicious behavior, threatrecognition module 316 to take into account a weighted scoring method todetermine whether an abuse is taking place and even to signal a type ofmitigation. These factors may each receive a different weight and theweighted values may be combined (e.g., by summing) to determine a score.If the score is above a threshold, the common abuse entity is identifiedas a potential abuser.

In addition, threat recognition module 316 may identify targets of theattack. To identify the target, threat recognition module 316 may lookto the destination addresses (e.g., IP addresses) of the packetsinvolved in the attack. In addition to identifying these destinationaddresses as targets of the attack, threat recognition module 316 mayalso aggregate the addresses into ranges and extrapolate otherdestinations that may be targeted using the techniques described abovefor identifying the source common abuse entity.

In response to the determination that the common abuse entity is apotential distributed malicious entity, mitigation module 124 looks tomitigation rules 308 to determine what action, if any, to take. Themitigation rules 308 may specify certain actions to take in depending oncharacteristics of the threat or the source. The characteristics of thethreat can include, for example, whether it is a rate-based abuse,whether it is an application abuse or denial of service abuse and how itwas identified by threat recognition module 316 (e.g., by geographicorigination, external threat feed, etc.)

When an abuse is detected, mitigation rules 308 can specify mitigationmodule to take one of several actions based on the characteristics ofthe abuse. First, mitigation module 124 sends a message to a specializedsoftware mitigation agent or network component, such as a firewall,router, switch, load balancer or DDOS mitigation appliance, to blocktraffic from addresses belonging to the common abuse entity. Second,mitigation module 124 can inform a DNS Bind9 Response Policy Zone tostop lookups of the DNS hostname or domain considered a threat. Third,mitigation module 124 can send a message to an application component toflag accounts the common abuse entity created to mark the accounts assuspicious. Fourth and finally, mitigation module 124 can send to anoperator an alert indicating the potential threat, allowing the operatorto decide what, if any, mitigating action to take.

Administrative interface module 318 may enable the operator to takeselect which mitigating action to take. Administrative interface module318 may be a web portal, command line interface (CLI) or API interfaceand also enable an operator to observe network data 302 and to specifyheuristics 304, threat rules 306, and mitigation rules 308.

When an operator takes an action on a potential threat, that action canbe used as feedback into machine learning module 330 for training. Thefeedback may be used to develop new mitigations rules 308. For example,after an operator manually mitigates a threat a certain number of times,a mitigation rule 308 may be created that automatically mitigates thethreat. In this way, by allowing feedback and modification of heuristics304, threat rules 306, and mitigation rules 308, administrativeinterface module 318 may enable a user to customize the abuse mitigationstrategy.

FIG. 4 is a flowchart illustrating a method 400 for abuse detection,according to an embodiment.

Method 400 begins at a step 402 by collecting data describing incomingrequests from a plurality of different external source addresses to theservice provider environment.

At step 404, the collected data is analyzed to compare the incomingrequests against a heuristic. For example, when the application callsare determined to match a regular expression, the requests aredetermined to be from the common abuse entity. When the incomingrequests are determined to match the heuristic, the requests aredetermined to be from a common abuse entity.

At step 406, the collected data is evaluated to determine that thecommon abuse entity is a potential network malicious entity of theservice provider environment. This determination may be made, forexample, based on the rate of incoming requests, based on an externalthreat feed, or other factors.

Finally, at step 408, the abuse is mitigated. This may involve sendingan alert to an operator, sending a message to a network component toblock traffic from addresses belonging to the common abuse entity, orsending a message to an application component to flag accounts thecommon abuse entity created to mark the accounts as suspicious.

CONCLUSION

Each of the devices and modules in FIGS. 1-3 may be implemented inhardware, software, firmware, or any combination thereof.

Each of the devices and modules in FIGS. 1-3 may be implemented on thesame or different computing devices. Such computing devices can include,but are not limited to, a personal computer, a mobile device such as amobile phone tablet device or laptop device, workstation, embeddedsystem, game console, television, set-top box, or any other computingdevice. Further, a computing device can include, but is not limited to,a device having a processor and memory, including a non-transitorymemory, for executing and storing instructions. The memory may tangiblyembody the data and program instructions. Software may include one ormore applications and an operating system. Hardware can include, but isnot limited to, a processor, a memory, and a graphical user interfacedisplay. The computing device may also have multiple processors andmultiple shared or separate memory components. For example, thecomputing device may be a part of or the entirety of a clustered ordistributed computing environment or server farm.

Identifiers, such as “(a),” “(b),” “(i),” “(ii),” etc., are sometimesused for different elements or steps. These identifiers are used forclarity and do not necessarily designate an order for the elements orsteps.

The present invention has been described above with the aid offunctional building blocks illustrating the implementation of specifiedfunctions and relationships thereof. The boundaries of these functionalbuilding blocks have been arbitrarily defined herein for the convenienceof the description. Alternate boundaries can be defined so long as thespecified functions and relationships thereof are appropriatelyperformed.

The foregoing description of the specific embodiments will so fullyreveal the general nature of the invention that others can, by applyingknowledge within the skill of the art, readily modify and/or adapt forvarious applications such specific embodiments, without undueexperimentation, without departing from the general concept of thepresent invention. Therefore, such adaptations and modifications areintended to be within the meaning and range of equivalents of thedisclosed embodiments, based on the teaching and guidance presentedherein. It is to be understood that the phraseology or terminologyherein is for the purpose of description and not of limitation, suchthat the terminology or phraseology of the present specification is tobe interpreted by the skilled artisan in light of the teachings andguidance.

The breadth and scope of the present invention should not be limited byany of the above-described exemplary embodiments, but should be definedonly in accordance with the following claims and their equivalents.

1. A computer-implemented method for detecting an abuse to a service provider environment, comprising: (a) collecting data describing incoming requests from a plurality of different external source addresses to the service provider environment; (b) analyzing the collected data to compare the incoming requests against a heuristic; (c) when the incoming requests are determined to match the heuristic, determining that the requests, having the plurality of different external source addresses, are from a common abuse entity; (d) evaluating the collected data to determine that the common abuse entity is a potential abuser of the service provider environment; in response to the determination that the common abuse entity is a potential abuser: (e) determining a target of the potential abuse; (f) selecting at least one of a plurality of mitigation devices in a network topology such that the selected mitigation device is upstream from the target; and (g) sending, to the selected mitigation device, a mitigation instruction to mitigate an attack from the common abuse entity.
 2. The method of claim 1, wherein the evaluating (d) comprises: (i) evaluating the collected data to determine a rate and type of incoming requests from the common abuse entity; (ii) determining whether the rate of incoming requests matches another heuristic; and (iii) when the rate of incoming requests is determined to match the other heuristic, determining that the common abuse entity is a potential abuser of the service provider environment.
 3. The method of claim 2, further comprising: (iv) aggregating collected data into counts of requests having common characteristics, wherein the evaluating comprises determining the rate based on the data aggregated in (iv).
 4. The method of claim 1, further comprising: (h) receiving, from an external threat feed, fingerprint data; and wherein the evaluating (d) comprises determining that the common abuse entity is the potential abuser based on the external data feed.
 5. The method of claim 4, wherein the receiving (h) comprises receiving, from an external threat feed, fingerprint data identifying a suspect source address, and further comprising: (i) determining that the suspect source address belongs to the common abuse entity, and wherein the evaluating (d) further comprises determining that the common abuse entity is the potential abuser based on whether the suspect source address from the external data feed belongs to the common abuse entity.
 6. The method of claim 1, further comprising: (h) in response to the determination that the common abuse entity is a potential abuser, sending a notification to an operator.
 7. The method of claim 1, further comprising: (h) in response to the determination that the common abuse entity is a potential distributed denial of service abuser, sending a message to a network component to block traffic from addresses belonging to the common abuse entity.
 8. The method of claim 1, further comprising: (h) in response to the determination that the common abuse entity is a potential abuser, sending a message to an application component to flag accounts the common abuse entity created to mark the accounts as suspicious.
 9. The method of claim 1, wherein the respective incoming requests are API calls and wherein the analyzing (b) comprises determining whether the API calls match a regular expression, and wherein the determining (c) comprises, when the API calls are determined to match a regular expression, determining that the requests, having a plurality of different external source addresses, are from the common abuse entity.
 10. The method of claim 1, wherein the heuristic has been user-defined in an administrative interface.
 11. The method of claim 1, wherein the at least one of the incoming requests includes an encrypted payload, further comprising: (h) prior to the collecting (a), decrypting, at a decryption server, the encrypted payload, wherein the collecting (a) comprises collecting data describing the payload and the evaluating (d) comprises evaluating the decrypted payload decrypted (e) to determine that the common abuse entity is a potential abuser; and (i) in response to the determination that the common abuse entity is a potential abuser, sending a mitigation instruction identifying a header of packets originating from a common abuse entity to a mitigation device, upstream from the decryption server, that mitigates abuses from encrypted packets.
 12. (canceled)
 13. The method of claim 1, further comprising wherein the determining (e) comprises determining a plurality of targets of the potential abuse, and wherein the selecting (f) comprises selecting a mitigation device from the plurality of mitigation devices such that the selected mitigation device is upstream from each of the plurality of targets.
 14. A system for detecting an abuse to a service provider environment, comprising: a computing device; a collection module, implemented on the computing device, that collects data describing incoming requests from a plurality of different external source addresses to the service provider environment; a source recognition module that (i) analyzes the collected data to compare the incoming requests against a heuristic, and, (ii) when a plurality of the incoming requests are determined to match the heuristic, determines that the requests, having the plurality of different external source addresses, are from a common abuse entity; a threat recognition module that evaluates the collected data to determine that the common abuse entity is a potential abuser of the service provider environment; and a mitigation module that, in response to the determination that the common abuse entity is a potential abuser, (i) determines a target of the potential abuse, (ii) selects at least one of a plurality of network components in a network topology such that the selected network component is upstream from the target, and (iii) sends, to the selected network component, a mitigation instruction to mitigate an attack from the common abuse entity.
 15. The system of claim 14, wherein the threat recognition module (i) evaluates the collected data to determine a rate of incoming requests from the common abuse entity, (ii) determines whether the rate of incoming requests exceeds a threshold, and (iii) when the rate of incoming requests is determined to exceed the threshold, determines that the common abuse entity is a potential abuser of the service provider environment.
 16. The system of claim 15, further comprising: an aggregation module that aggregates the collected data into counts of requests having common characteristics, wherein the evaluating comprises determining the rate based on the aggregated data.
 17. The system of claim 14, wherein the threat recognition module receives, from an external threat feed, fingerprint data identifying a suspect source address and determines that the common abuse entity is the potential abuser based on whether the suspect source address from the external data feed belongs to the common abuse entity.
 18. The system of claim 14, further comprising: a mitigation module that, in response to the determination that the common abuse entity is a potential distributed abuser, sends a notification to an operator.
 19. (canceled)
 20. The system of claim 14, further comprising: a mitigation module that, in response to the determination that the common abuse entity is a potential distributed denial of service abuser, sending a message to an application component to flag accounts the common abuse entity created to mark the accounts as suspicious.
 21. The system of claim 14, wherein the respective incoming requests are API calls and wherein the source recognition module determines whether the API calls match a regular expression, and, when the API calls are determined to match a regular expression, determines that the requests, having a plurality of different external source addresses, are from the common abuse entity.
 22. The system of claim 14, wherein the heuristic has been user defined in an administrative interface.
 23. The system of claim 14, wherein the threat recognition module determines whether the common abuse entity is a potential abuser based on: (i) a number of domains the common abuse entity hosts; (ii) a geographic origination of the source addresses; (iii) a reputation data of the source addresses; or (iv) whether data has been detected as associated with prior malicious activity.
 24. A program storage device tangibly embodying a program of instructions executable by at least one machine to perform a method for detecting abuse, said method comprising: (a) collecting data describing incoming requests from a plurality of different external source addresses to the service provider environment; (b) analyzing the collected data to compare the incoming requests against a heuristic; (c) when the incoming requests are determined to match the heuristic, determining that the requests, having the plurality of different external source addresses, are from a common abuse entity; (d) evaluating the collected data to determine that the common abuse entity is a potential abuser of the service provider environment; in response to the determination that the common abuse entity is a potential abuser: (e) determining a target of the potential abuse; (f) selecting at least one of a plurality of mitigation devices in a network topology such that the selected mitigation device is upstream from the target; and (g) sending, to the selected mitigation device, a mitigation instruction to mitigate an attack from the common abuse entity.
 25. The method of claim 4, wherein the fingerprint data is at least one of a reputation of a source address, DNS data, protocol information, geographic location, or application layer expressions and transactions.
 26. The method of claim 1, further comprising: (h) in response to the determination that the common abuse entity is a potential abuser, sending a message to mitigate an infection with a malware or virus. 